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Program Description 1 

Many students, even those who are academically well prepared, 
arrive to college without the strong study skills and coping strategies 
required to effectively navigate the learning and social environments 
in higher education. First year experience courses, often referred to 
as college success courses or freshman seminars, are courses for 
first-year students in 2-year and 4-year colleges. As reported by the 
National Resource Center for The First-Year Experience and Students 
in Transition, first year experience courses are required for all first- 
year students at approximately 52% of 4-year institutions. 2 Further- 
more, 87% of 4-year institutions that offer freshman seminars do so 
for academic credit. 3 While they are not required for all students, first 
year experience courses are frequently required for special popula- 
tions (e.g., academically underprepared students, students in specific 
majors, and students taking remedial courses). The general goals 
of first year experience courses are to support the academic perfor- 
mance, social development, persistence, and degree completion of 
college students. 4 Additionally, first year experience courses often aim 
to increase students’ sense of campus community and connection 
to their institutions, while giving students the opportunity to interact 
with faculty and peers. 5 While courses vary in terms of content and focus, most first year experience courses are 
designed to introduce students to “campus resources, time management, study skills, career planning, cultural 
diversity, and student development issues.” 6 
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This intervention report presents findings 
from a systematic review of first year 
experience courses conducted using 
the WWC Procedures and Standards 
Handbook, version 3.0, and the 
Supporting Postsecondary Success 
review protocol, version 3.0. 


Research 7 

The What Works Clearinghouse (WWC) identified four studies of first year experience courses that both fall within 
the scope of the Supporting Postsecondary Success topic area and meet WWC group design standards. No stud- 
ies meet WWC group design standards without reservations, and four studies meet WWC group design standards 
with reservations. Together, these studies included 12,091 freshman college students in four colleges across the 
United States. 


The WWC considers the extent of evidence for first year experience courses for freshman college students to be 
medium to large for one outcome domain— credit accumulation. The WWC considers the extent of evidence for first 
year experience courses for freshman college students to be small for two outcome domains— degree attainment 
(college) and general academic achievement (college). There were no studies that meet WWC design standards in the 
other domains specified as eligible in the review protocol, specifically college access and enrollment, attendance, and 
labor market outcomes. This intervention report does not report on the effectiveness of first year experience courses 
for those three domains. (See the Effectiveness Summary on p. 6 for more details of effectiveness by domain.) 
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Effectiveness 

First year experience courses were found to have potentially positive effects on credit accumulation, degree attain- 
ment (college), and general academic achievement (college) for freshman college students. 


Table 1. Summary of findings 8 




Improvement index (percentile points) 




Outcome domain 

Rating of effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

Credit accumulation 

Potentially positive effects 

+12 

+4 to +14 

3 

11,455 

Medium to large 

Degree attainment 
(college) 

Potentially positive effects 

+9 

na 

1 

8,290 

Small 

General academic 

achievement 

(college) 

Potentially positive effects 

+6 

na 

1 

1,869 

Small 


na = not applicable 
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Program Information 

Background 

The first freshman seminar was offered in the late 1880s. After experiencing a lull in popularity, the seminar was 
reintroduced in the 1970s for three primary reasons: 9 (a) an increased number of diverse students were coming to 
college unprepared for the academic and social challenges; (b) entering students had to sift through a vast number 
of new issues (such as curricular choices) upon matriculation because the enterprise of higher education was more 
complex; and (c) institutions realized that their reliance on peers to provide support was insufficient for students’ 
success. In the early 1990s, Barefoot and Fidler 10 identified five types of first year experience courses that still thrive 
on college campuses today. These are extended orientation, academic seminars with uniform content, academic 
seminars with variable content, introduction to a discipline or professional seminars, and basic study skills semi- 
nars. More recently, hybrid versions of the courses, which include elements from more than one type, have also 
been offered. 

The most common type of course is the extended orientation seminar. Its primary goal is to support student suc- 
cess during the freshman year. The most frequent topics associated with these seminars are study skills, campus 
resources, time management, career exploration, campus policies, and academic advising. Academic seminars 
with uniform or variable content are increasing in number. 11 The content of these seminars can be uniform (i.e., 
each section has a set curriculum) or variable (i.e., different sections are built around a specific topic and/or the 
expertise of the faculty member). All academic seminars also focus on developing students’ writing, critical think- 
ing, and study skills. Seminars for introducing students to a discipline or profession focus on preparing students 
for the demands of their major and eventual profession. Basic study skills seminars are more circumscribed and 
revolve around helping students prepare for the academic rigors of college by focusing on building skills such as 
time management and note-taking. 12 

For information about seminars intended for students taking remedial courses, please refer to the intervention 
report of first year experience courses for students in developmental education. 13 

Program details 

Three of the first year experience courses reviewed in this report were of the extended orientation type (Clouse, 
2012; Shoemaker, 1995; Wilkerson, 2009). Course topics varied, but all were designed to ease students’ transition 
to college and provide an extended orientation to the respective colleges. One of the first year experience courses 
reviewed in this report was academic with variable content (Jamelske, 2009). The sections of this course were 
embedded in a number of core courses at the university (e.g., biology, economics, psychology) but also included 
a number of elements common in extended orientation. 


Cost 

Jamelske (2009) indicated that the fall 2006 budget for the first year experience program at the medium-sized 
midwestern university was $237,700. During the fall semester of 2006, the university offered 85 variable-content 
academic seminars, which enrolled about 20 students each. The first year experience courses were taught by 
62 different instructors that semester. No information was provided regarding what expenses were included in 
this budget figure. No other studies included cost information. 
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Research Summary 

The WWC identified 97 eligible studies that investigated the effects 
of first year experience courses for first-year college students. An 
additional 176 studies were identified but do not meet WWC eligibility 
criteria for review in this topic area. Citations for all 273 studies are in 
the References section, which begins on p. 8. 

The WWC reviewed 97 eligible studies against group design standards. 

No studies are randomized controlled trials that meet WWC group design standards without reservations, and four 
studies (Clouse, 2012; Jamelske, 2009; Shoemaker, 1995; Wilkerson, 2008) use quasi-experimental designs that 
meet WWC group design standards with reservations. Those four studies are summarized in this report. Ninety- 
three studies do not meet WWC group design standards. 

Summary of studies meeting WWC group design standards without reservations 

No studies of first year experience courses met WWC group design standards without reservations. 

Summary of studies meeting WWC group design standards with reservations 

Clouse (2012) conducted a quasi-experimental evaluation of a non-compulsory first year experience course at a 
large public university in the western United States. The intervention group included 3,754 first-time, first-semester 
freshmen who enrolled in the credit-bearing seminar between 1995 and 2005. The first year experience course was 
designed to help freshmen at the university transition from high school to college and included topics in the follow- 
ing three areas: academic, personal, and community. Academic components included study, communication, and 
technology skills; personal components included a focus on developing personal goals and responsibilities; and 
community components focused on developing relationships and learning opportunities outside the classroom. 
Each section generally included 15 or fewer students and was taught by one of several faculty members and one 
junior teaching assistant. The comparison condition was comprised of 4,536 first-time, first-semester freshmen at 
the same university who did not enroll in the first year experience course. These students had access to the typi- 
cal college resources. Baseline equivalence for this study’s outcomes was established using data provided by the 
author to the WWC. 

Jamelske (2009) conducted a quasi-experimental evaluation of a first year experience course at a medium-sized 
public university in the midwestern United States. The intervention group consisted of 1 ,456 first-time freshmen 
who enrolled in one of 85 different first year experience sections in the fall of 2006. The university in the study 
offers 100 different course sections, and a majority of first-time freshmen (> 85%) enrolls in one of these sections. 
The first year experience courses are linked to a specific course topic (e.g., biology, economics, psychology), are 
capped at 20 students, and include small group peer activities, close work with a faculty member, and an upper- 
classman as a student peer mentor. The program goals are to introduce students to core course content, enhance 
academic skills (including content knowledge, study skills, time management, etc.), strengthen connections to the 
university, engage students in out-of-class activities, and enhance student accountability. The comparison condi- 
tion was comprised of 413 first-time freshmen who did not enroll in the first year experience course. These students 
enrolled in courses typical of first-year students at the university. 

Shoemaker (1995) conducted a quasi-experimental evaluation of a first year experience course at the University of 
California, Irvine campus in 1993. The intervention group consisted of 34 students who enrolled in the two-credit 
course called “The University Experience: Issues and Options for Unaffiliated/Undecided Students” in the fall quar- 
ter of 1993. The course met twice weekly and included guest lectures from faculty and professional staff, discus- 
sion, and assignments designed to promote a successful transition to college. Course topics covered library skills, 


Table 2. Scope of reviewed research 


Grades 

Postsecondary 

Delivery method 

Whole class 

Program type 

Practice 
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interacting with faculty, university policies and procedures, learning styles, goal setting, cultural diversity, and stress 
and time management skills. The comparison condition included 602 unaffiliated or undeclared freshmen who did 
not enroll in the seminar. 

Wilkerson (2008) conducted a quasi-experimental study of a first year experience course at the University of Texas 
at San Antonio. The intervention group was comprised of 133 first-time freshmen who enrolled in a first year 
experience course at the university from 2002 to 2004. The course was a core three-credit class for students in the 
College of Social and Behavioral Sciences. No other details about the content of the course were provided by the 
author. The comparison condition was comprised of 1 ,163 freshman students who did not enroll in the first year 
experience course. 
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Effectiveness Summary 

The WWC review of first year experience courses for the Supporting Postsecondary Success topic area includes 
outcomes in six domains: credit accumulation, degree attainment (college), general academic achievement (col- 
lege), college access and enrollment, college attendance, and labor market. The four studies of first year experience 
courses that meet WWC group design standards reported findings in three of the six domains: (a) credit accumula- 
tion, (b) degree attainment (college), and (c) general academic achievement (college). The findings below present 
the authors’ estimates and WWC-calculated estimates of the size and statistical significance of the effects of first 
year experience courses on freshman college students. For a more detailed description of the rating of effective- 
ness and extent of evidence criteria, see the WWC Rating Criteria on p. 39. 

Summary of effectiveness for the credit accumulation domain 


Table 3. Rating of effectiveness and extent of evidence for the credit accumulation domain 


Rating of effectiveness 

Criteria met 

Potentially positive effects 

Evidence of a positive effect with no 
overriding contrary evidence. 

In two of the three studies that reported findings, the estimated impact of the intervention on outcomes in the 
credit accumulation domain was positive and statistically significant. In the remaining study, there was no statisti- 
cally significant impact of the intervention on the single outcome in the credit accumulation domain. 

Extent of evidence 

Criteria met 

Medium to large 

Two studies that included 9,586 students in two universities reported evidence of effectiveness in the credit 
accumulation domain. 


Three studies that meet WWC group design standards with reservations reported findings in the credit accumulation 
domain. 

Clouse (2012) reported on students’ persistence at the university to the third semester (i.e., whether students com- 
pleted their freshman year). The author reported, and the WWC confirmed, that there was a positive and statistically 
significant difference in retention between students who participated in the first year experience course and those 
who did not. The WWC characterizes this finding as a statistically significant positive effect. 

Jamelske (2009) reported on students’ retention at the university after 1 year. The author reported that there was no 
statistically significant difference between first year experience course participants and comparison participants on 
retention. The WWC characterizes this finding as an indeterminate effect. 

Wilkerson (2009) reported on students’ retention at the university to the second year (i.e., whether students returned 
for their second year fall semester). The author reported, and the WWC confirmed, that there was a positive and 
statistically significant difference between first year experience course participants and comparison participants on 
retention. The WWC characterizes this finding as a statistically significant positive effect. 

Thus, for the credit accumulation domain, two studies that met WWC group design standards with reservations 
showed statistically significant positive effects, while one study that met WWC group design standards with res- 
ervations showed an indeterminate effect. This results in a rating of potentially positive effects, with a medium to 
large extent of evidence. 
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Summary of effectiveness for the degree attainment (college) domain 

Table 4. Rating of effectiveness and extent of evidence for the degree attainment (college) domain 


Rating of effectiveness 

Criteria met 

Potentially positive effects 

Evidence of a positive effect with 
no overriding contrary evidence. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the degree 
attainment (college) domain was positive and statistically significant. 

Extent of evidence 

Criteria met 

Small 

One study that included 8,290 students in one university reported evidence of effectiveness in the degree 
attainment (college) domain. 


One study that meets WWC group design standards with reservations reported findings in the degree attainment 
(college) domain. 

Clouse (2012) reported on students’ attainment of an undergraduate degree at their home institution and at any 
US college participating in the National Student Clearinghouse Data Program. The author reported, and the WWC 
confirmed, that there was a statistically significant difference between first year experience course participants and 
comparison participants on undergraduate degree attainment at the home institution and a statistically significant 
difference between the two groups in undergraduate degree attainment at any US college. The WWC characterizes 
the mean effect across the two measures in this domain as a statistically significant positive effect. 

Thus, for the degree attainment (college) domain, one study showed a statistically significant positive effect. This 
results in a rating of potentially positive effects, with a small extent of evidence. 

Summary of effectiveness for the general academic achievement (college) domain 


Table 5. Rating of effectiveness and extent of evidence for the general academic achievement (college) domain 


Rating of effectiveness 

Criteria met 

Potentially positive effects 

Evidence of a positive effect with 
no overriding contrary evidence. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the general 
academic achievement (college) domain was positive and statistically significant. 

Extent of evidence 

Criteria met 

Small 

One study that included 1,869 students in one university reported evidence of effectiveness in the general 
academic achievement (college) domain. 


Two studies that meet WWC group design standards with reservations reported findings in the general academic 
achievement (college) domain. 

Jamelske (2009) reported on students’ cumulative grade point average after 1 year. The author reported, and 
the WWC confirmed, that there was a statistically significant difference between first year experience course 
participants and comparison participants on cumulative grade point average. The WWC characterizes this finding 
as a statistically significant positive effect. 

Shoemaker (1995) reported on students’ grade point average for the spring quarter of 1994 for the students who 
participated in the first year experience course in the fall quarter of 1993 and the students who did not participate 
in the seminar. The statistical significance and the effect size for the comparison that meets WWC group design 
standards with reservations could not be determined from the information reported by the author. 

Thus, for the general academic achievement (college) domain, one study showed a statistically significant positive 
effect. This results in a rating of potentially positive effects, with a small extent of evidence. 
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study does not meet WWC group design standards because equivalence of the analytic intervention and 
comparison groups prior to the intervention was necessary and not demonstrated. 

Helfrich, C. (1999). The effects of a student orientation course on student retention and success (Doctoral disserta- 
tion). Available from ProQuest Dissertations and Theses database. (UMI No. 9939146) The study does not 
meet WWC group design standards because equivalence of the analytic intervention and comparison groups 
prior to the intervention was necessary and not demonstrated. 

Hendel, D. D. (2001 , April). The relative contribution of participating in a first-year seminar on student satisfaction 
and retention into the sophomore year. Paper presented at the annual meeting of the American Educational 
Research Association, Seattle, WA. http://files.eric.ed.gov/fulltext/ED453724.pdf. The study does not meet 
WWC group design standards because equivalence of the analytic intervention and comparison groups prior 
to the intervention was necessary and not demonstrated. 

Additional source: 

Hendel, D. D. (2006-2007). Efficacy of participating in a first-year seminar on student satisfaction and reten- 
tion. Journal of College Student Retention, 8(4), 413-423. 

Hoff, M. R, Cook, D., & Price, C. (1994, February). Five years of DCS 101: An empirical analysis. Paper presented 
at the annual Freshman Year Experience Conference. http://files.eric.ed.gov/fulltext/ED369362.pdf. The study 
does not meet WWC group design standards because equivalence of the analytic intervention and comparison 
groups prior to the intervention was necessary and not demonstrated. 

Hoff, M. R, Cook, D., & Price, C. (1996). The first five years of freshman seminars at Dalton College: Student success 
and retention. Journal of the Freshman Year Experience, 8(2), 33-42. The study does not meet WWC group 
design standards because equivalence of the analytic intervention and comparison groups prior to the inter- 
vention was necessary and not demonstrated. 

Horstman, W. A. (2004). A quantitative comparison of two different college success course formats on freshmen at 
a four-year college (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI 
No. 3160094) The study does not meet WWC group design standards because equivalence of the analytic 
intervention and comparison groups prior to the intervention was necessary and not demonstrated. 
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House, J. D., & Kuchynka, S. J. (1997). The effects of a freshman orientation course on the achievement of health 
sciences students. Journal of College Student Development, 38(5), 540-542. The study does not meet WWC 
group design standards because equivalence of the analytic intervention and comparison groups prior to the 
intervention was necessary and not demonstrated. 

Isaacson, M. L. (2008). Mattering and persistence relative to a first-year seminar for residential students at a Great 
Plains university (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI 
No. 3333958) The study does not meet WWC group design standards because equivalence of the analytic 
intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Johnson, M. H. (2012). An analysis of retention factors in undergraduate degree programs in science, technology, 
engineering, and mathematics (Doctoral dissertation). Available from ProQuest Dissertations and Theses data- 
base. (UMI No. 3518473) The study does not meet WWC group design standards because equivalence of the 
analytic intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Kaplan, N. S. (2000). The impact of a College Experience 101 course on student retention, academic success, and 
academic performance for at-risk freshmen at St. John’s University (Doctoral dissertation). Available from 
ProQuest Dissertations and Theses database. (UMI No. 9967703) The study does not meet WWC group 
design standards because equivalence of the analytic intervention and comparison groups prior to the inter- 
vention was necessary and not demonstrated. 

Keenan, K., & Gabovitch, R. (1995, October). Evaluating the impact of a freshman seminar program on student 
development and retention. Paper presented at the meeting of the North East Association for Institutional 
Research, Burlington, VT. The study does not meet WWC group design standards because equivalence of the 
analytic intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Keup, J. R. (2005-2006). The impact of curricular interventions on intended second-year re-enrollment. Journal of 
College Student Retention, 7(1-2), 61-89. The study does not meet WWC group design standards because 
equivalence of the analytic intervention and comparison groups prior to the intervention was necessary and 
not demonstrated. 

Klein, B. J. (2013). Building student connections: A successful first-year experience course and community college 
retention (Unpublished doctoral dissertation). Iowa State University, Ames. The study does not meet WWC 
group design standards because equivalence of the analytic intervention and comparison groups prior to the 
intervention was necessary and not demonstrated. 

Klimovich, D. E. (1994). Effect of academic support services on retention of college freshmen: A study of persisters 
and nonpersisters (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI 
No. 9509750) The study does not meet WWC group design standards because equivalence of the analytic 
intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Klinkenberg, L. B. (2013). A quantitative analysis of a mandatory student success course on first-time full-time 

student college academic progress and persistence (Unpublished doctoral dissertation). Iowa State University, 
Ames. The study does not meet WWC group design standards because the measures of effectiveness could 
not be attributed solely to the intervention. 

Kolb, M. M. (2005). The relationship between state appropriations and student retention at public, four-year institu- 
tions of higher education (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. 
(UMI No. 3159880) The study does not meet WWC group design standards because equivalence of the 
analytic intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Kronenberger, J. L. (2012). Student success: The effects of a community college first-year course (Unpublished doc- 
toral dissertation). University of Dayton, OH. The study does not meet WWC group design standards because 
equivalence of the analytic intervention and comparison groups prior to the intervention was necessary and 
not demonstrated. 
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Lang, D. J. (2007). The impact of a first-year experience course on the academic performance, persistence, and 
graduation rates of first-semester college students at a public research university. Journal of the First-Year 
Experience and Students in Transition, 79(1), 9-25. The study does not meet WWC group design standards 
because equivalence of the analytic intervention and comparison groups prior to the intervention was neces- 
sary and not demonstrated. 

Lehmuller, P. A. (201 0). The effect of multiple interventions on freshman college student engagement and reten- 
tion (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3439267) 
The study does not meet WWC group design standards because equivalence of the analytic intervention and 
comparison groups prior to the intervention was necessary and not demonstrated. 

Magun-Jackson, S. (1 996). The relationship of a freshman orientation course to minority and non-minority retention 
at a large urban university (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. 
(UMI No. 9705696) The study does not meet WWC group design standards because equivalence of the ana- 
lytic intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Maw, J. A. (2005). The facilitation of student success: Incorporating affective, behavioural, and cognitive factors 
into first-year experience programs (Unpublished doctoral dissertation). University of Manitoba, Winnipeg. 

The study does not meet WWC group design standards because equivalence of the analytic intervention and 
comparison groups prior to the intervention was necessary and not demonstrated. 

McGrath, S. M. (2011). Success course intervention for students on academic probation in science majors: A 
longitudinal quantitative examination of the treatment effects on performance, persistence, and graduation 
(Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3453501) The 
study does not meet WWC group design standards because the measures of effectiveness could not be 
attributed solely to the intervention. 

McGrath, S. M., & Burd, G. D. (2012). A success course for freshmen on academic probation: Persistence and 

graduation outcomes. NACADA Journal, 32(1), 43-52. The study does not meet WWC group design standards 
because the measures of effectiveness could not be attributed solely to the intervention. 

McMillan-Haron, V. (2003). The impact of freshman success courses on freshman-to-sophomore persistence and 
academic achievement at WICHE Urban University and College (Doctoral dissertation). Available from ProQuest 
Dissertations and Theses database. (UMI No. 3091806) The study does not meet WWC group design stan- 
dards because equivalence of the analytic intervention and comparison groups prior to the intervention was 
necessary and not demonstrated. 

Merica, M. M. (2014). Applying propensity score analysis in the evaluation of a first-year seminar program (Doctoral 
dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3621124) The study does 
not meet WWC group design standards because equivalence of the analytic intervention and comparison 
groups prior to the intervention was necessary and not demonstrated. 

Micceri, T., & Wajeeh, E. (1 999, June). Evidence supporting a university experience course’s efficacy at a metro- 
politan university, and associated effective processes. Paper presented at the meeting of the Association for 
Institutional Research, Seattle, WA. The study does not meet WWC group design standards because equiva- 
lence of the analytic intervention and comparison groups prior to the intervention was necessary and not 
demonstrated. 

Miller, J. W., & Janz, J. C. (2007). The retention impact of a first-year seminar on students with varying pre-college 
academic performance. Journal of the First-Year Experience and Students in Transition, 79(1), 47-62. The 
study does not meet WWC group design standards because equivalence of the analytic intervention and 
comparison groups prior to the intervention was necessary and not demonstrated. 
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Additional source: 

Janz, J. C. (2004). The retention impact of the freshman seminar on students with varying pre-college academic 
performance (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI 
No. 3141095) 

Minchella, D. J., Yazvac, C. W., Fodrea, R. A., & Ball, G. (2002). Biology resource seminar: First aid for the first 

year. The American Biology Teacher, 64(5), 352-357. The study does not meet WWC group design standards 
because equivalence of the analytic intervention and comparison groups prior to the intervention was neces- 
sary and not demonstrated. 

Nguyen, Q. T. H. (2013). Modeling completion at a community college (Doctoral dissertation). Available from Pro- 
Quest Dissertations and Theses database. (UMI No. 1523327) The study does not meet WWC group design 
standards because equivalence of the analytic intervention and comparison groups prior to the intervention 
was necessary and not demonstrated. 

Odell, P. M. (1996). Avenues to success in college: A non-credit eight-week freshman seminar. Journal of the Fresh- 
man Year Experience, 8(2), 79-92. The study does not meet WWC group design standards because equiva- 
lence of the analytic intervention and comparison groups prior to the intervention was necessary and not 
demonstrated. 

Odom, J. E. (1994). Encouraging growth in moral judgement: A cognitive-developmentally based first year orienta- 
tion program (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 
9425483) The study does not meet WWC group design standards because equivalence of the analytic inter- 
vention and comparison groups prior to the intervention was necessary and not demonstrated. 

Outlaw, J. S. (2008). Academic outcomes of academic success programs (Doctoral dissertation). Available from 
ProQuest Dissertations and Theses database. (UMI No. 3319077) The study does not meet WWC group 
design standards because equivalence of the analytic intervention and comparison groups prior to the inter- 
vention was necessary and not demonstrated. 

Owen, J. (2012). Predictors of student degree attainment at a four-year, mid-sized, public university in the Great 

Plains (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3507929) 
The study does not meet WWC group design standards because equivalence of the analytic intervention and 
comparison groups prior to the intervention was necessary and not demonstrated. 

Pietropaolo, P. A. (1994). The effects of an academic intervention on persistence of science/technology majors at a 
community college (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI 
No. 9522551) The study does not meet WWC group design standards because equivalence of the analytic 
intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Potts, G. T., & Schultz, B. L. (2008). The freshman seminar and academic success of at-risk students. College Student 
Journal, 42(2), 647-658. The study does not meet WWC group design standards because the measures of 
effectiveness could not be attributed solely to the intervention. 

Rosa, K. J. (1997). Evaluation of a commercial student retention program at Topeka Technical College (Unpublished 
doctoral dissertation). Nova Southeastern University, Fort Lauderdale, FL. The study does not meet WWC 
group design standards because the measures of effectiveness could not be attributed solely to the intervention. 

Ryan, M. R, & Glenn, P. A. (2004). What do first-year students need most: Learning strategies instruction or academic 
socialization? Journal of College Reading and Learning, 34(2), 4-28. http://files.eric.ed.gov/fulltext/EJ690847. 
pdf. The study does not meet WWC group design standards because equivalence of the analytic intervention 
and comparison groups prior to the intervention was necessary and not demonstrated. 

Schnell, C. A., & Doetkott, C. D. (2002-2003). First year seminars produce long-term impact. Journal of College 
Student Retention, 4(4), 377-391 . The study does not meet WWC group design standards because equiva- 
lence of the analytic intervention and comparison groups prior to the intervention was necessary and not 
demonstrated. 
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Schnell, C. A. J. (2000). First year seminar effectiveness: Success, retention, and subsequent graduation of college 
freshmen (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 
9961010) The study does not meet WWC group design standards because equivalence of the analytic inter- 
vention and comparison groups prior to the intervention was necessary and not demonstrated. 

Additional source: 

Schnell, C. A., Louis, K. S., & Doetkott, C. (2003). The first-year seminar as a means of improving college 
graduation rates. Journal of the First-Year Experience and Students in Transition, 75(1), 53-76. 

Sheldon, C. Q., & Durdella, N. R. (2009). Examining the relationship between freshman seminars, student achieve- 
ment, and persistence: A study of first-time Santa Monica College students enrolled in Counseling 20. 
Unpublished manuscript, Santa Monica College, CA. The study does not meet WWC group design standards 
because equivalence of the analytic intervention and comparison groups prior to the intervention was neces- 
sary and not demonstrated. 

Sidle, M. W., & McReynolds, J. (1999). The freshman year experience: Student retention and student success. 
NASPA Journal, 36(4), 288-300. The study does not meet WWC group design standards because equiva- 
lence of the analytic intervention and comparison groups prior to the intervention was necessary and not 
demonstrated. 

Simmons, G., Wallins, J., & George, A. (1995). The effects of a freshman seminar on at-risk under-, over-, and low 
achievers. NACADA Journal, 75(1), 8-14. The study does not meet WWC group design standards because 
equivalence of the analytic intervention and comparison groups prior to the intervention was necessary and 
not demonstrated. 

Starke, M. C., Harth, M., & Sirianni, F. (2001). Retention, bonding, and academic achievement: Success of a first- 
year seminar. Journal of the First-Year Experience and Students in Transition, 73(2), 7-35. The study does not 
meet WWC group design standards because equivalence of the analytic intervention and comparison groups 
prior to the intervention was necessary and not demonstrated. 

Stewart, B. W. (1997). The effects of a freshman seminar course on student retention, academic success, and 

academic performance (Unpublished doctoral dissertation). Mississippi State University, Starkville. The study 
does not meet WWC group design standards because equivalence of the analytic intervention and compari- 
son groups prior to the intervention was necessary and not demonstrated. 

Stovall, M. L. (1 999). Relationships between participation in a community college student success course and aca- 
demic performance and persistence (Doctoral dissertation). Available from ProQuest Dissertations and Theses 
database. (UMI No. 9945007) The study does not meet WWC group design standards because equivalence of 
the analytic intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Strayhorn, T. L. (2009). An examination of the impact of first-year seminars on correlates of college student reten- 
tion. Journal of the First-Year Experience and Students in Transition, 27(1), 9-27. The study does not meet 
WWC group design standards because equivalence of the analytic intervention and comparison groups prior 
to the intervention was necessary and not demonstrated. 

Sullivan, C. J. (201 0). Academic self-regulation, academic performance, and college adjustment: What is the 
first-year experience for college students? (Doctoral dissertation). Available from ProQuest Dissertations 
and Theses database. (UMI No. 3422866) The study does not meet WWC group design standards because 
equivalence of the analytic intervention and comparison groups prior to the intervention was necessary and 
not demonstrated. 

Summerlee, A., & Murray, J. (2010). The impact of enquiry-based learning on academic performance and student 
engagement. Canadian Journal of Higher Education, 40(2), 78-94. http://files.eric.ed.gov/fulltext/EJ904276. 
pdf. The study does not meet WWC group design standards because equivalence of the analytic intervention 
and comparison groups prior to the intervention was necessary and not demonstrated. 

Tampke, D. R., & Durodoye, R. (2013). Improving academic success for undecided students: A first-year seminar/ 
learning community approach [Study 1: FYE vs. comparison]. Learning Communities Research and Practice, 
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7(2), Article 3. The study does not meet WWC group design standards because equivalence of the analytic 
intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Tampke, D. R., & Durodoye, R. (2013). Improving academic success for undecided students: A first-year seminar/ 
learning community approach [Study 2: FYE+LC vs. comparison]. Learning Communities Research and Prac- 
tice, 7(2), Article 3. [Study 2: FYE+LC vs. comparison] The study does not meet WWC group design standards 
because equivalence of the analytic intervention and comparison groups prior to the intervention was neces- 
sary and not demonstrated. 

Tebbe, C. M. (2007). The effectiveness of a learning strategies course on college student-athletes’ and non-athletes’ 
adjustment, academic performance, and retention after the first two years of college (Doctoral dissertation). 
Available from ProQuest Dissertations and Theses database. (UMI No. 3300977) The study does not meet 
WWC group design standards because equivalence of the analytic intervention and comparison groups prior 
to the intervention was necessary and not demonstrated. 

teDuits, D. J. (2008). The effects of a first-year academic support program on first time in college freshmen (Doctoral 
dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3289810) The study does 
not meet WWC group design standards because equivalence of the analytic intervention and comparison 
groups prior to the intervention was necessary and not demonstrated. 

Thachil, S. A. (2013). Factors that relate to the persistence of first-generation undergraduate students in a pub- 
lic university (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 
3585038) The study does not meet WWC group design standards because the measures of effectiveness 
could not be attributed solely to the intervention. 

Vancil, J. J. (2001). The development, design, course content, and student outcomes of a freshman orientation sem- 
inar (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 1406273) 
The study does not meet WWC group design standards because equivalence of the analytic intervention and 
comparison groups prior to the intervention was necessary and not demonstrated. 

Walleri, R. D., Stoker, C. L., & Stoering, J. (1997, May). Building a community of learning: A comprehensive 
approach to assisting at-risk students. Paper presented at the meeting of the Association for Institutional 
Research, Orlando, FL. The study does not meet WWC group design standards because equivalence of the 
analytic intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Walls, L. L. (2000). The impact of a first-year orientation course on at-risk students at a large, public research univer- 
sity (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 9988482) 
The study does not meet WWC group design standards because equivalence of the analytic intervention and 
comparison groups prior to the intervention was necessary and not demonstrated. 

Williford, A. M., Chapman, L. C., & Kahrig, T. (2000-2001). The university experience course: A longitudinal study 
of student performance, retention, and graduation. Journal of College Student Retention: Research, Theory & 
Practice, 2(4), 327-340. The study does not meet WWC group design standards because equivalence of the 
analytic intervention and comparison groups prior to the intervention was necessary and not demonstrated. 

Yale, A. A. (1 999). The impact of a one-credit freshman seminar on student retention, academic progress, and 
academic and social integration, while controlling for the volunteer effect (Doctoral dissertation). Available 
from ProQuest Dissertations and Theses database. (UMI No. 9957798) The study does not meet WWC group 
design standards because equivalence of the analytic intervention and comparison groups prior to the inter- 
vention was necessary and not demonstrated. 

Yockey, F. A., & George, A. A. (1998). The effects of a freshman seminar paired with supplemental instruction. Journal 
of the First-Year Experience and Students in Transition, 70(2), 57-76. The study does not meet WWC group 
design standards because equivalence of the analytic intervention and comparison groups prior to the inter- 
vention was necessary and not demonstrated. 
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Studies that are ineligible for review using the Supporting Postsecondary Success Evidence Review Protocol 

Abernathy, L. T. (1 997). The impact of variations in the psychosocial orientation of first-year college students on 
academic and social integration (Doctoral dissertation). Available from ProQuest Dissertations and Theses 
database. (UMI No. 1388088) The study is ineligible for review because it is out of scope of the protocol. 

Abts, M. (2012). Effectiveness of online community college success courses (Doctoral dissertation). Available from 
ProQuest Dissertations and Theses database. (UMI No. 3502982) The study is ineligible for review because 
it does not use an eligible design. 

Aldredge, T. W. (1 997). The impact of a college success course on students’ long-term academic persistence (Doctoral 
dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 9821953) The study is 
ineligible for review because it is out of scope of the protocol. 

Anderson, E. G. (2008). Freshmen retention: The impact of a first year experience program on student satisfaction 
and academic performance (Doctoral dissertation). Available from ProQuest Dissertations and Theses data- 
base. (UMI No. 3348688) The study is ineligible for review because it does not use an eligible design. 

Andrade, M. S. (2006). A first-year seminar for international students. Journal of The First-Year Experience and Stu- 
dents in Transition, 78(1), 85-103. The study is ineligible for review because it does not use an eligible design. 

Andrade, M. S. (2008-2009). The value of a first-year seminar: International students’ insights in retrospect. Journal 
of College Student Retention: Research, Theory & Practice, 10(4), 483-506. The study is ineligible for review 
because it does not use an eligible design. 

Barefoot, B. O., Warnock, C. L., Dickinson, M. R, Richardson, S. E., & Roberts, M. R. (1998). Exploring the evi- 
dence: Reporting outcomes of first-year seminars. The first-year experience, vot. 2 (Monograph Series No. 25). 
Columbia: National Resource Center for the First- Year Experience, University of South Carolina. The study is 
ineligible for review because it does not use an eligible design. 

Barnes, J. (2012). The first-year experience impact on student success in developmental education. Journal of 
Applied Research in the Community College, 20(1), 27-35. The study is ineligible for review because it does 
not use a sample aligned with the protocol. 

Bement, S. A. (201 0). The effects of first-year experience course on students’ grades in remedial english and 
mathematics courses (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. 

(UMI No. 3433121) The study is ineligible for review because it does not use a sample aligned with the protocol. 

Beran, P. B. (1996). A comparison of the effects of major and interest specific orientation courses versus generic 
orientation courses (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI 
No. 9718294) The study is ineligible for review because it does not use an eligible design. 

Berkovitz, R. A., & O’Quin, K. (2006-2007). Predictors of graduation of readmitted “at risk” college students. Jour- 
nal of College Student Retention: Research, Theory & Practice, 8(2), 199-214. The study is ineligible for review 
because it is out of scope of the protocol. 

Bianchino, E. G. G. (1994). The impact of a community college freshman orientation program on student persis- 
tence rate and cumulative college grade point average (Doctoral dissertation). Available from ProQuest Disser- 
tations and Theses database. (UMI No. 3426145) The study is ineligible for review because it does not use an 
eligible design. 

Borden, V. M. H., Burton, K. L., Evenbeck, S. E., & Williams, G. A. (1997). The impact of academic support pro- 
grams on student performance and persistence. Research Brief, 4(4), 1-16. The study is ineligible for review 
because it is out of scope of the protocol. 

Boroch, D., & Hope, L. (2009). Effective practices for promoting the transition of high school students to college: 

A review of literature with implications for California Community College practitioners. Sacramento: Center for 
Student Success, Research and Planning Group of the California Community Colleges, http://files.eric.ed.gov/ 
fulltext/ED51 9286.pdf. The study is ineligible for review because it is not a primary analysis of the effect of an 
intervention. 
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Bradfute, R. W. (2011). Community college counseling and mathematics study skills: Retention and success (Doctoral 
dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3472675) The study is 
ineligible for review because it does not use a sample aligned with the protocol. 

Brennan, J. B. (2008). College and career effectiveness skills and their impact on student success and learning 
(Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3338142) The 
study is ineligible for review because it does not use an eligible design. 

Brown, L. J. E. (1994). Interventions for academic success of African American freshman college students (Doctoral 
dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 9519872) The study is 
ineligible for review because it is out of scope of the protocol. 

Brown-Durham, G. (2006). The effectiveness of the BICUM Study-Reading Instructional Strategy on reading com- 
prehension and self-efficacy levels of first-year, first-semester students enrolled in a three-credit college 
developmental reading and study skills course (Doctoral dissertation). Available from ProQuest Dissertations 
and Theses database. (UMI No. 3221500) The study is ineligible for review because it does not use a sample 
aligned with the protocol. 

Buchanan, S. R. (1993). University 101 for high school students. Journal of the Freshman Year Experience, 5(2), 
49-68. The study is ineligible for review because it does not use a sample aligned with the protocol. 

Burgert, J. C. (1 999). The role of a study skills course on improving the academic success of “at risk’’ college fresh- 
men (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 9928055) 
The study is ineligible for review because it is out of scope of the protocol. 

Burley, H. (1994, May). Persistence: A meta-analysis of college developmental studies programs. Paper presented 
at the meeting of the Association for Institutional Research, New Orleans, LA. The study is ineligible for review 
because it is not a primary analysis of the effect of an intervention. 

Candia, P. P. (1 998). An investigation of the effects of orientation and study skills courses on retention and grade 
point average of first time in college students in a community college in south Texas (Doctoral dissertation). 
Available from ProQuest Dissertations and Theses database. (UMI No. 9915214) The study is ineligible for 
review because it does not use a sample aligned with the protocol. 

Cardin, L. H. L. (2001). Effects of individualized web-based instructional modules on the strategic learning strategies 
of students enrolled in a university study skills course (Doctoral dissertation). Available from ProQuest Disser- 
tations and Theses database. (UMI No. 3033770) The study is ineligible for review because it does not use an 
eligible design. 

Carraway, J. D. (2006). High school to college transition programs (Doctoral dissertation). Available from ProQuest 
Dissertations and Theses database. (UMI No. 3208836) The study is ineligible for review because it does not 
use an eligible design. 

Chasteen, B. C. (2005). A new student orientation program: Its relationship to retention and academic performance 
(Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3189905) The 
study is ineligible for review because it is out of scope of the protocol. 

Cone, A. L. (1991). Sophomore academic retention associated with a freshman study skills and college adjustment 
course. Psychological Reports, 69, 312-314. The study is ineligible for review because it is out of scope of the 
protocol. 

Crissman, J. L. (1999). The impact of clustering freshman seminars with English composition courses on new stu- 
dents’ grade point average and retention rates (Unpublished doctoral dissertation). The Pennsylvania State 
University, State College. The study is ineligible for review because it does not use an eligible design. 
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Appendix A.1: Research details for Clouse (2012) 

Clouse, W. A. (2012). The effects of non-compulsory freshman seminar and core curriculum completion 
ratios on post-secondary persistence and baccalaureate degree attainment (Doctoral disserta- 
tion). Available from ProQuest Dissertations and Theses database. (UMI No. 3523633) 

Table A1. Summary of findings Meets WWC group design standards with reservations 


Study findings 
Average improvement index 

Outcome domain Sample size (percentile points) Statistically significant 


Degree attainment (college) 8,290 students +9 Yes 

Credit accumulation 8,290 students +8 Yes 


Setting 

The study was conducted at a large public university in the western United States using archi- 
val data extracted from the student records central information warehouse. Data from all first- 
time, full-time incoming freshmen enrolled in the fall semester from 1995 to 2005 were eligible 
for analysis. 

Study sample 

The author reported baseline characteristics on the full sample. The mean age was 18.69 
(SD = 1 .42). Of the full sample, 60% were male and 88% were White. The average amount of 
need-based aid received over the first year was $1 ,035.74 (SD = $2,471 .44). 

Intervention 

group 

Students in the intervention condition chose to enroll in a credit-bearing freshman seminar 
course designed for first-time, first-semester freshman college students. The first year experi- 
ence course was designed to help freshmen at the university transition from high school to 
college and included topics in the following three areas: academic, personal, and community. 
Academic components included study, communication, and technology skills; personal com- 
ponents included a focus on developing personal goals and responsibilities; and community 
components focused on developing relationships and learning opportunities outside the class- 
room. Each section generally included 15 or fewer students and was taught by one of several 
faculty members and one junior teaching assistant. 

Comparison 

group 

This study utilized a business-as-usual comparison condition. The comparison group con- 
sisted of those students who chose not to enroll in the freshman seminar. These students 
had access to typical college resources. Services received by the comparison group were 
not monitored. 

Outcomes and 
measurement 

The author reported three dichotomous outcome variables: (1) Home degree conferred 
(whether a student received an undergraduate degree from their home institution within 6 
years of entering); (2) External degree conferred (whether a student received an undergraduate 
degree from any US college participating in the National Student Clearinghouse Data Program 
within 6 years of initial postsecondary enrollment); and, (3) Persistence (whether students per- 
sisted at the home institution from the first to the third semester, i.e., completed the freshman 
year). All data were obtained from either the study site's student records central information 
warehouse or the National Student Clearinghouse. 
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Support for Implementation of the freshman seminar program was initiated by the college in 1993. The 
implementation course is taught by 2-4 faculty and/or instructional staff. These faculty/staff are supported by 
teaching assistants who are selected from currently enrolled students. No information on train- 
ing for instructors is reported. 


Appendix A.2: Research details for Jamelske (2009) 

Jamelske, E. (2009). Measuring the impact of a university first-year experience program on student 


GPA and retention. Higher Education, 57(3), 373-391. 

Table A2. Summary of findings Meets WWC group design standards with reservations 


Outcome domain 

Sample size 

Study findings 

Average improvement index 
(percentile points) 

Statistically significant 

General academic 

1,869 students 

+6 

Yes 

achievement (college) 




Credit accumulation 

1,869 students 

+4 

No 


Setting The study took place at a medium-sized public university in the midwestern United States. A 
total of 1 00 first year experience course sections were offered to students in the fall of 2006. 

Study sample In the overall sample of students, 40% were male, 93% were White, the average age was 18.6 
(SD = .34) years, 16% were from low-income households, 42% were first generation college 
students, and the average ACT score was 24.3 (SD = 2.87). 


Intervention The first-year experience (FYE) program in this study was designed to offer academic and 

group extracurricular services to first-time freshmen and to help integrate participating students into 

the university setting. The university offers 100 different FYE course sections, and almost all 
(> 85%) first-time freshmen enroll in an FYE course. The FYE courses are linked to a specific 
course topic (e.g., biology, economics, algebra, calculus, psychology, Spanish) and involve 
small group peer activities (FYEs were capped at 20 students for enrollment), close work with a 
faculty member, and a student peer mentor. The FYE program goals are to introduce students 
to core course content, enhance academic skills (including content knowledge, study skills, 
time management, etc.), strengthen connections to the university, engage students in out-of- 
class activities, and enhance student accountability. The FYE also aims to foster connections 
with at least one faculty member and provide a peer educational learning community. Regard- 
less of whether teaching an FYE or non-FYE course, instructors must cover the same basic 
topical materials (i.e., both FYE and non-FYE biology must cover the same biology content), 
and grades must be determined on comparable assignments and exams. The author noted that 
integrating the FYE material without compromising the original core course was difficult, and 
that the time commitment for extracurricular activities was problematic for FYE faculty. 


Comparison 

group 


Students in the comparison condition were full-time, first-year college students who did not 
enroll in any FYE course sections during fall 2006. These students enrolled in courses typical 
of first-year students at the university. 
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Outcomes and 
measurement 

The study reports findings for two eligible outcomes: (1) retention: a binary measure indicating 
whether students were retained at the university for the fall 2007 semester after being enrolled 
for their entire freshman year in 2006-07; and (2) cumulative grade point average (GPA), a 
continuous measure representing students’ cumulative GPA during their first academic year in 
2006-07. No information is provided about the source of the outcome data in this study. 

Support for 
implementation 

Informal, voluntary faculty and peer mentor training workshops were provided, with no proce- 
dures to hold instructors accountable for meeting program goals. There was no formal appli- 
cation process for selecting FYE faculty and no added value placed on performance reviews 
for teaching FYE. 


Appendix A.3: Research details for Shoemaker (1995) 

Shoemaker, J. S. (1995, April). Evaluating the effectiveness of extended orientation for new, undecided 
freshmen. Paper presented at the meeting of the American Educational Research Association, 

San Francisco, CA. http://files.eric.ed.gov/fulltext/ED384303.pdf. 

Table A3. Summary of findings Meets WWC group design standards with reservations 


Study findings 
Average improvement index 

Outcome domain Sample size (percentile points) Statistically significant 


Outcome domain 

Study findings 

Average improvement index 

Sample size (percentile points) Statistically significant 

General academic 
achievement (college) 

636 students na na 

na = not applicable 


Setting 

The freshman orientation course was offered at the University of California, Irvine campus in 

1993. The students included all incoming first-year students with undeclared majors during 
that academic year. 

Study sample 

The intervention group was 53% female, and the comparison group was 49% female. The 
sample was racially diverse. Students in the intervention group were 68% Asian, 18% White, 
and 14% other minority. Students in the comparison group were 56% Asian, 24% other minor- 
ity, and 20% White. Of the full study sample, 24% of the intervention group and 25% of the 
comparison group were classified as low income, and 9% of the intervention group and 8% of 
the comparison group were English language learners. 

Intervention 

group 

The intervention was a two-credit freshman orientation course called “The University Experi- 
ence: Issues and Options for Unaffiliated/Undecided Students.” The course met twice weekly 
and was comprised of guest lectures from faculty and professional staff, group discussion, 
and various assignments intended to ease the transition to college. Course topics covered 
library skills, interacting with faculty, university policies and procedures, assessing individual 
learning styles, setting goals, cultural diversity, stress management skills, and opportunities for 
student leadership. 

Comparison 

group 

The comparison condition included all unaffiliated or undeclared freshmen who did not enroll 
in the orientation course. These students received traditional college services and coursework; 
they did not participate in the orientation course. 
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Outcomes and 
measurement 

The study included five outcomes in two domains relevant for the review protocol. In the credit 
accumulation domain: retention rates (enrolled at the end of the first year), academic progress 
by units (normal with 36 or more units vs. subject to probation/disqualification with fewer than 

36 units), and quarter units completed were reported. In the general academic achievement 
(college) domain: GPA (college GPA), and Writing status (satisfied vs. not satisfied completion 
of entrance requirement in Writing) were reported. Only the GPA outcome met WWC group 
design standards. The study also included one outcome not relevant to the review protocol. 
Student satisfaction was measured using student-reported ratings of lecture and discus- 
sion sessions on a 5-point scale and by asking students whether they would recommend the 
course to other freshmen. 

Support for 
implementation 

No information was provided regarding support for implementation. 


Appendix A.4: Research details for Wilkerson (2009) 

Wilkerson, S. L. (2008). An empirical analysis of factors that influence the first year to second year 

retention of students at one large, Hispanic Serving Institution (HSI) (Doctoral dissertation). Avail- 
able from ProQuest Dissertations and Theses database. (UMI No. 3333787) 

Table A4. Summary of findings Meets WWC group design standards with reservations 


Study findings 
Average improvement index 

Outcome domain Sample size (percentile points) Statistically significant 


Outcome domain 

Study findings 

Average improvement index 

Sample size (percentile points) Statistically significant 

Credit accumulation 

1,296 students 14 Yes 

Setting 

The study took place at the University of Texas at San Antonio. The university is a metropolitan, 
comprehensive public university that offers bachelor’s, master’s, and doctoral degree programs. 

Study sample 

Characteristics of the full sample are reported. The sample contained 55.3% females and 

44.7% males. Of the sample, 46.1 % were Hispanic, 39.8% were White, 7.2% were African 
American, 6.7% were Asian or Pacific Islander, and 0.2% were American Indian or Alaskan 

Native. In addition, 75% of the students were financial aid applicants, and 48.8% were first 
generation students. 

Intervention 

group 

The first-year seminar was offered as a three-credit course and was considered a core require- 
ment for students in the College of Social and Behavioral Sciences. No specific information on 
how the course is implemented is reported. 

Comparison 

group 

The comparison group for the study is considered a business-as-usual comparison; these 
students did not participate in the first-year seminar. No information on monitoring this group 
is reported. 
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Outcomes and 
measurement 


Support for 
implementation 


One eligible outcome was reported in the study: first-to-second year retention. The university’s 
Office of Institutional Research manages the student databases, including the BANNER sys- 
tem, which houses admission, academic, demographic, and personal information for students. 
Data were collected from this system to identify seminar participants and to assess retention 
into the second year. Retention was measured as a dichotomous outcome. 

No information was provided regarding support for implementation. 
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Appendix B: Outcome measures for each domain 


Credit accumulation 

Enrollment 

Credit accumulation was measured by student enrollment in fall 2007 after being enrolled the entire 2006-07 
academic year (as cited in Jamelske, 2009). 

First-to-second year retention 

Retention into the second year at the university was obtained from the university’s Office of Institutional 

Research BANNER system (as cited in Wilkerson, 2009). 

Persistence from first to third semester 

Persistence (whether a student persisted at their home institution from their first to their third semester) was 
obtained from the study site's student records central information warehouse (as cited in Clouse, 2012). 

Degree attainment (college) 

Degree conferred at home institution 

Degree attainment (college) was measured by 6-year graduation rates at the institution in the study. Data were 
obtained from the central information warehouse at the study site (as cited in Clouse, 2012). 

Degree conferred at any institution 

Degree attainment (college) was measured by 6-year graduation rates at any college or university participating 
in the National Student Data Clearinghouse data program. Data were obtained from the National Student 
Clearinghouse (as cited in Clouse, 2012). 

General academic 
achievement (college) 

Cumulative grade point average (GPA) 

General academic achievement (college) was assessed by cumulative GPA at the end of the spring semester of 
the 2007 school year (as cited in Jamelske, 2009). 

GPA 

General academic achievement (college) was measured with student GPAs at the end of the 1994 spring quarter 
(as cited in Shoemaker, 1995). 
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Appendix C.1: Findings included in the rating for the credit accumulation domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Clouse, 2012 a 

Persistence from first to third 
semester 

College 

students 

8,290 

nr 

nr 

nr 

0.21 

+8 

.00 


Domain average for credit accumulation (Clouse, 2012) 0.21 +8 Statistically 


significant 



First-to-second year retention College 1,296 nr 

students 

nr 

nr 

0.36 

+14 

.00 

Domain average for credit accumulation (Wilkerson, 2009) 



0.36 

+14 

Statistically 

significant 

Domain average for credit accumulation across all studies 



0.23 

+12 

na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 
an average individual's percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the domain average was determined by the WWC. Some 
statistics may not sum as expected due to rounding, na = not applicable, nr = not reported. 

a For Clouse (201 2), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported in the 
original study. The WWC effect size was calculated by taking the natural log of the reported odds ratio and dividing the result by 1 .65 (Cox logit method). This study is characterized as 
having a statistically significant positive effect because the effect for at least one measure within the domain is positive and statistically significant, and no effects are negative and 
statistically significant, accounting for multiple comparisons. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 

b For Jamelske (2009), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported in 
the original study. This study is characterized as having indeterminate effects because the reported effect size was neither statistically significant nor large enough to be substantially 
important. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 

c For Wilkerson (2009), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported 
in the original study. This study is characterized as having a statistically significant positive effect because the effect for at least one measure within the domain is positive and 
statistically significant, and no effects are negative and statistically significant, accounting for multiple comparisons. For more information, please refer to the WWC Procedures and 
Standards Handbook (version 3.0), p. 26. 
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Appendix C.2: Findings included in the rating for the degree attainment (college) domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Clouse, 2012 a 

Degree conferred at home 
institution 

College 

students 

8,290 

nr 

nr 

nr 

0.23 

+9 

.00 

Degree conferred at any 

College 

8,290 

nr 

nr 

nr 

0.24 

+9 

.00 


institution students 

Domain average for degree attainment (college) (Clouse, 2012) 0.24 +9 Statistically 

significant 


Domain average for degree attainment (college) across all studies 0.24 +9 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 
an average individual's percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC. 
Some statistics may not sum as expected due to rounding, na = not applicable, nr = not reported. 

a For Clouse (201 2), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The p-values presented here 
were reported in the original study. The WWC effect size was calculated by taking the natural log of the reported odds ratio and dividing the result by 1 .65 (Cox logit method). This study 
is characterized as having a statistically significant positive effect because the effect for at least one measure within the domain is positive and statistically significant, and no effects are 
negative and statistically significant, accounting for multiple comparisons. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 


Appendix C.3: Findings included in the rating for the general academic achievement (college) domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Jamelske, 2009 a 

Cumulative grade point 
average 

College 

students 

1,869 

nr 

nr 

nr 

0.14 

+6 

.01 


Domain average for general academic achievement (college) (Jamelske, 2009) 0.15 +6 Statistically 

significant 


Shoemaker, 1995 b 

Grade point average College 636 2.79 2.74 0.05 na na na 

students (0.50) (0.59) 

Domain average for general academic achievement (college) (Shoemaker, 1995) na na na 


Domain average for general academic achievement (college) across all studies 0.15 +6 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the 
comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are given the 
intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an average indi- 
vidual’s percentile rank that can be expected if the individual is given the intervention. Some statistics may not sum as expected due to rounding, na = not applicable, nr = not reported. 
a For Jamelske (2009), no corrections for clustering or multiple comparisons and no difference-in-differences adjustment were needed. The p-value presented here was reported in 
the original study. The WWC effect size was calculated using the 0LS regression coefficient and the pooled standard deviation. This study is characterized as having a statistically 
significant positive effect because the effect for at least one measure within the domain is positive and statistically significant, and no effects are negative and statistically significant, 
accounting for multiple comparisons. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 

b For Shoemaker (1 995), an effect size could not be computed from the information provided in the study, though the study used an approved method of analyzing the impact of the 
intervention; the means shown in the table were not adjusted for baseline covariates, and were therefore not used to compute an effect size. The findings for this study cannot be 
characterized by the WWC because effect sizes could not be computed. 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 

Study rating 

Criteria 

Meets WWC group design 
standards without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 

Meets WWC group design 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 

standards with reservations 

attrition that has established equivalence of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statistically significant positive effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 
of studies show indeterminate effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 

The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 

The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Intervention 
Intervention report 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 


Single-case design 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p. 39. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

An educational program, product, practice, or policy aimed at improving student outcomes. 

A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an interven- 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 39. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 
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Glossary of Terms 


Standard deviation The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance Statistical significance is the probability that the difference between groups is a result of 

chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 


Substantively important a substantively important finding is one that has an effect size of 0.25 or greater, regardless 

of statistical significance. 

Systematic review a review of existing literature on a topic that is identified and reviewed using explicit meth- 
ods. A WWC systematic review has five steps: 1) developing a review protocol; 2) searching 
the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high quality studies and their find- 
ings; 4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Development Services Group under contract ED-IES-12-C-0084. 
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